Impact of Biases in the False-Positive Rate on Null Hypothesis Testing

To test whether a prediction algorithm has any true predictive power, it is necessary to compare its performance with that expected under various well-defined null hypotheses. These null hypotheses must include the assumption that the prediction algorithm lacks any true predictive power. In the context of studies investigating the predictability of epileptic seizures, two approaches have been introduced for this purpose: analytical performance estimates (Schelter et al. 2006a; Snyder et al. 2008; Winterhalder et al. 2003; Wong et al. 2007) and seizure predictor surrogates based on constrained randomizations of the original seizure predictor (Andrzejak et al. 2003; Kreuz et al. 2004). We recently extended the Monte Carlo framework of seizure predictor surrogates by introducing the concept of alarm times surrogates (Andrzejak et al. 2009). This new type of seizure predictor surrogates is as straightforward to implement as seizure time surrogates (Andrzejak et al. 2003), while offering the same flexibility as measure profile surrogates (Kreuz et al. 2004). Therefore, alarm times surrogates combine the advantages of existing seizure predictor surrogates and remedy some of their shortcomings. The resulting flexibility in formulating distinct, testable null hypotheses is the key advantage of alarm times surrogates over analytical performance estimates. In our previous work (Andrzejak et al. 2009), we illustrated this by using artificial seizure time sequences and artificial original seizure predictors specifically constructed to be consistent or inconsistent with various null hypotheses. This allowed us to determine the frequency of null hypothesis rejections obtained from the analytical performance estimate and different types of alarm times surrogates under controlled conditions. The flexibility of alarm times surrogates was illustrated by including postseizure nonstationarities in the null hypothesis, which is not possible for analytical performance estimates. Furthermore, we showed that for Poisson predictors, which fulfill the null hypothesis of analytical performance estimates, the frequency of false-positive null hypothesis rejections substantially exceeds the significance level for long mean interalarm intervals, revealing an intrinsic bias of these estimates. Finally, we showed that alarm times surrogates and analytical performance estimates have a similar high statistical power to recognize seizure predictors with true predictive power. We extend this previous work here by illustrating the robustness of alarm times surrogates coNteNts